Entropy and Margin Maximization for Structured Output Learning
نویسندگان
چکیده
We consider the problem of training discriminative structured output predictors, such as conditional random fields (CRFs) and structured support vector machines (SSVMs). A generalized loss function is introduced, which jointly maximizes the entropy and the margin of the solution. The CRF and SSVM emerge as special cases of our framework. The probabilistic interpretation of large margin methods reveals insights about margin and slack rescaling. Furthermore, we derive the corresponding extensions for latent variable models, in which training operates on partially observed outputs. Experimental results for multiclass, linear-chain models and multiple instance learning demonstrate that the generalized loss can improve accuracy of the resulting classifiers.
منابع مشابه
Maximum Entropy Discrimination Markov Networks
Standard maximum margin structured prediction methods lack a straightforward probabilistic interpretation of the learning scheme and the prediction rule. Therefore its unique advantages such as dual sparseness and kernel tricks cannot be easily conjoined with the merits of a probabilistic model such as Bayesian regularization, model averaging, and ability to model hidden variables. In this pape...
متن کاملRegularized Structured Output Learning with Partial Labels
We consider the problem of learning structured output probabilistic models with training examples having partial labels. Partial label scenarios arise commonly in web applications such as taxonomy (hierarchical) classification, multi-label classification and information extraction from web pages. For example, label information may be available only at the internal node level (not at the leaf le...
متن کاملOnline Relative Margin Maximization for Statistical Machine Translation
Recent advances in large-margin learning have shown that better generalization can be achieved by incorporating higher order information into the optimization, such as the spread of the data. However, these solutions are impractical in complex structured prediction problems such as statistical machine translation. We present an online gradient-based algorithm for relative margin maximization, w...
متن کاملJoint Maximum Margin and Maximum Entropy Learning of Graphical Models
INFERRING structured predictions based on correlated covariates remains a central problem in many fields, including NLP, computer vision, and computational biology. Typically, both the input covariates and output predictions can be high-dimensional, multi-modal, noisy, partially observable, and bearing latent structures, each of these characteristics adds a degree of complexity to the task of l...
متن کاملMaximum Entropy: A Special Case of Minimum Cross-entropy Applied to Nonlinear Estimation by an Artificial Neural Network
The application of cross-entropy information processing optimizations to artificial neural network (ANN) training can provide decreased sensitivity to accelerated learning rates as well as insights into the information processing structure of the network. In order to assess the cross-entropy between the desired training goal and the evolving state of network information at each training step, t...
متن کامل